TRUST Review: Ai2 Asta
Information Tool Reviews Framework VerdictTR Transparency ●●●3/3 | RE Reliability ●●○2.5/3 | US Usability ●●○2.5/3 | SE Soundness ●●○2/3 | TC Traceability ●●●3/3 | Overall ●●○2.6/3 |
|---|
An open-sourced and well documented tool that shows how it works and what it does in a transparant manner. The results are often quite good, and always come with reasoning/evidence. The fact that the tool verbosely interprets your request and breaks it down into specific requirements for inclusion is very helpful even without any results as a help to fine-tune (re)search questions.
This feels like a nice tool for multiple use cases, as the transparacy in the process makes it suitable for e.g. learning how LLM's interpret your prompts and decide relevancy. It is also relatively simple and open source, so it can make a fine base for a more complex tooling.
- Transparent LLM usage
- Clearly supported results
- Open source and free
- Natural language interaction works well
- Only semantic scholar as a backend
- Limited filtering / metadata available in this interface
- No direct connections with other tools like Zotero
- Cannot use existing papers as input
- No way to do combine natural language queries with manual/deterministic additions
- Base privacy/content policy does not match our requirements, need to self-host to circumvent
Detailed Scores
Quality Gates
- PS1PASS
Vendor must explicitly state that user search queries, history, and uploaded content are not sold, shared with third-party advertisers, or used for commercial profiling. User data must not be retained beyond the stated service period without explicit consent.
No reviewer note
PS1DetailsRequirement
Vendor must explicitly state that user search queries, history, and uploaded content are not sold, shared with third-party advertisers, or used for commercial profiling. User data must not be retained beyond the stated service period without explicit consent.
Background
Academic users frequently input sensitive or unpublished research material into search tools. Even non-AI tools can violate privacy standards by tracking user queries, selling search-trend data to third-party advertisers, or retaining search history without consent. A clear data privacy policy is essential for trust in academic contexts regardless of whether the tool uses AI. Look for this statement in the privacy policy, terms of service, or a dedicated data governance page. N/A is applicable only if evaluating a purely local, offline-installed tool that has no network connection and cannot transmit user inputs (e.g., a locally compiled desktop script).
Examples
- pass
- The tool's privacy policy states: 'User search queries, browsing history, and uploaded documents are never shared with third-party advertisers, sold to data brokers, or used for commercial profiling. Data is retained only for the duration needed to provide the service and is deleted within 30 days of account closure.' This is confirmed on the privacy policy page dated within the last 12 months.
- fail
- The terms of service state that 'user interaction data may be shared with select partners for improved service delivery' or there is no mention of third-party data sharing practices at all. Alternatively, the tool's privacy policy allows selling aggregated search data to third parties.
- na
- Evaluating a locally compiled desktop search tool that runs entirely offline, has no network connection, and cannot transmit user inputs to any server.
- PS2—
Vendor must explicitly state that user queries/inputs/uploads are NOT used to train future AI models.
While the terms clearly state they can and will use your input to train models, the tool is also available fully open source and can be self-hosted without sending any data to Ai2 or others. However, when using their hosted version, this would be marked as 'Fail'.
PS2DetailsRequirement
Vendor must explicitly state that user queries/inputs/uploads are NOT used to train future AI models.
Background
Academic users frequently input sensitive or unpublished research material into AI-powered search tools. If a tool uses these inputs to train its models, it could leak proprietary research, pre-publication findings, or confidential institutional data. A clear opt-out or non-training policy is essential for trust in academic contexts. Look for this statement in the privacy policy, terms of service, or a dedicated AI ethics page. This gate applies only to AI-powered tools.
Examples
- pass
- The tool's documentation includes a section titled 'Data and AI Training' that states: 'User queries, uploaded documents, and search interactions are never used to train, fine-tune, or improve our AI models.' This is confirmed on the privacy policy page dated within the last 12 months.
- fail
- The terms of service state that 'interactions with the service may be used to improve our products and services' with no option to opt out, or there is no mention of training data practices at all.
- na
- Evaluating a traditional (non-AI) citation database like PubMed or Web of Science that uses keyword matching and Boolean search without any machine learning model.
Reviewer note
While the terms clearly state they can and will use your input to train models, the tool is also available fully open source and can be self-hosted without sending any data to Ai2 or others. However, when using their hosted version, this would be marked as 'Fail'.
- IP1PASS
The vendor's Terms of Service must explicitly state that the user retains full copyright and intellectual property rights over all uploaded documents, prompts, and generated outputs. The vendor must not claim any license to distribute, publish, or commercially exploit user uploads.
No reviewer note
IP1DetailsRequirement
The vendor's Terms of Service must explicitly state that the user retains full copyright and intellectual property rights over all uploaded documents, prompts, and generated outputs. The vendor must not claim any license to distribute, publish, or commercially exploit user uploads.
Background
Academic researchers routinely upload unpublished manuscripts, grant proposals, proprietary datasets, and pre-publication findings into search or synthesis tools. Some tools state in their Terms of Service that they acquire a perpetual, royalty-free license to use uploaded content for any purpose, including model improvement, redistribution, or commercial exploitation. This creates a direct risk of intellectual property loss, which can compromise patent applications, embargoed research, and institutional data ownership policies. Review the Terms of Service, paying particular attention to sections on 'User Content', 'License Grant', or 'Intellectual Property'. N/A may apply for tools that do not accept any user-uploaded content, for example, a read-only citation database with no document upload, prompt, or file submission features. If the tool accepts any user content, this gate should be evaluated.
Examples
- pass
- The Terms of Service include a section titled 'Your Content — Ownership' that states: 'You retain all rights, title, and interest in and to your content. We do not claim any ownership or license to distribute, publish, or commercially exploit your uploaded documents, prompts, or outputs.' No contradictory language appears elsewhere in the agreement.
- fail
- The Terms of Service state: 'By uploading content, you grant us a worldwide, non-exclusive, royalty-free license to use, reproduce, modify, and distribute your content for the purpose of improving our services.' Alternatively, the Terms make no mention of user intellectual property rights at all.
- na
- Evaluating a read-only citation database (e.g., Google Scholar) that accepts no user-uploaded documents, prompts, or files; there is no user content for the vendor to claim rights over.
- AC1PASS
Tool must support keyboard navigation to all major features, allow text resizing to at least 200% without content loss, and provide alt text for images. An accessibility statement or conformance claim (e.g., WCAG 2.1 AA) is strong supporting evidence.
While it does not pass the full scan, it does match most of WCAG. The found issues are mostly trivial to fix with some additional labeling.
AC1DetailsRequirement
Tool must support keyboard navigation to all major features, allow text resizing to at least 200% without content loss, and provide alt text for images. An accessibility statement or conformance claim (e.g., WCAG 2.1 AA) is strong supporting evidence.
Background
Accessibility is a legal requirement in many jurisdictions (EU Accessibility Act, Section 508 in the US) and an ethical imperative for tools used in publicly funded institutions. Academic tools must be usable by people with visual, motor, or cognitive disabilities. Reviewers should test keyboard-only navigation, zoom the browser to 200%, and check whether images have meaningful alt text. An accessibility statement demonstrates institutional commitment. N/A is not applicable for web-based tools, which must meet accessibility standards. It may be considered for backend command-line tools or raw APIs with no user interface.
Examples
- pass
- You can navigate to all search features, settings, and export options using only the Tab and Enter keys. Zooming the browser to 200% preserves the layout with no overlapping text or hidden controls. The tool's footer links to an accessibility statement declaring WCAG 2.1 AA conformance.
- fail
- Critical features such as the search filters or export button cannot be reached via keyboard; they only respond to mouse clicks. Zooming to 200% causes the navigation menu to overflow and become partially hidden.
- na
- Evaluating a backend REST API with no graphical user interface, consumed exclusively through programmatic calls; there are no visual controls, images, or layouts to evaluate for accessibility compliance.
Reviewer note
While it does not pass the full scan, it does match most of WCAG. The found issues are mostly trivial to fix with some additional labeling.
Transparency●●●3/3
- TR13
Complete list of indexed sources with coverage dates and update frequency.
They make use of their own database, Semantic Scholar, which is regularly updated.
TR1DetailsData source clarity
Background
Source transparency is the first step toward reproducible research. When a tool discloses which databases, publishers, and indices it searches, users can assess whether its coverage is adequate for their discipline and identify potential blind spots. Without this information, researchers cannot evaluate whether the tool's results are comprehensive or skewed toward particular sources. Look for a dedicated 'Sources' or 'Coverage' page, or documentation within the tool's help section. Note: this question evaluates macro-level data scope transparency, understanding the overall boundaries and composition of the tool's index. It is distinct from TC2 (Source quality indicators), which evaluates item-level metadata enrichment such as retraction warnings and publication type labels on individual search results. This question is not applicable if the tool does not retrieve or aggregate information from external databases, publishers, or indices, functioning solely as a platform for users to search their own internally uploaded or generated content.
Examples
- 0
- The tool's website and help documentation make no mention of which databases or publishers it searches. A 'Sources' page does not exist, and support inquiries return vague responses like 'we search the best academic sources.'
- 1
- The tool says it covers 'peer-reviewed journals, conference proceedings, and patents' but does not name any specific databases (e.g., Scopus, IEEE Xplore, PubMed) or indicate the scope of coverage.
- 2
- The documentation lists Scopus, Web of Science, IEEE Xplore, and PubMed as indexed sources. It mentions 'additional regional databases' without naming them or specifying coverage dates.
- 3
- A dedicated 'Data Sources' page lists 47 databases with coverage periods (e.g., 'PubMed: 1950–present', 'arXiv: 1991–present') and a 'Last updated' timestamp showing the list was refreshed within the past month.
Reviewer note
They make use of their own database, Semantic Scholar, which is regularly updated.
Evidence (2)
Capture2026-06-19 12:09 
Capture2026-06-19 12:10 - TR23
The tool explicitly names the retrieval method, ranking or filtering steps, generation model, and any post-processing.
The full source code is available if required to dig into every detail of how the tool works.
TR2DetailsMethodology disclosure
Background
Understanding how a tool processes queries and generates results is essential for interpreting its outputs. This question evaluates whether the tool explains its underlying technology: retrieval method, ranking algorithm, and any AI generation steps. In academic contexts, researchers need to know whether results are purely retrieved from indexed databases or synthesized by a language model, as this affects how they interpret and cite the output. Reviewers do not need to verify the mathematics of the algorithm. Look for a 'System Architecture', 'Technical White Paper', or 'How It Works' link in the footer or help documentation. If the vendor explains their retrieval, ranking, and generation steps in plain language (even without specific model parameters), they qualify for a higher score. This question only applies to AI-powered tools. Mark this question as N/A if the resource is a traditional, static database or non-digital reference material that does not utilize any algorithmic search, ranking, or AI generation components.
Examples
- 0
- The tool's 'How it works' page simply says 'Our advanced technology finds the best results for your research.' No technical documentation, white papers, or methodology descriptions exist anywhere on the site.
- 1
- The 'Features' webpage states: 'Powered by AI' and 'Uses machine learning to deliver relevant results.' Despite these claims, the documentation completely lacks specific technical details, omitting any mention of the exact models, algorithms, or data sources used to rank the output.
- 2
- The documentation states: 'We use a BERT-based semantic search model to retrieve relevant passages from indexed literature, combined with a GPT-4 summarization layer for synthesized answers.' The retrieval and ranking process is described at a conceptual level.
- 3
- A technical white paper details the full pipeline: (1) query expansion using fine-tuned T5, (2) hybrid BM25 + dense retrieval from a 200M passage index, (3) cross-encoder re-ranking, (4) GPT-4 answer generation with chain-of-thought prompting, and (5) post-processing citation verification against the retrieved passages.
Reviewer note
The full source code is available if required to dig into every detail of how the tool works.
Evidence (2)
Capture2026-06-19 12:16 
Capture2026-06-19 12:13
Reliability●●○2.5/3
- RE13
All claims are verifiable against original sources across every test query; no fabricated citations, no misattributions, and synthesis accurately represents areas of consensus and ongoing debate.
No reviewer note
RE1DetailsAccuracy and hallucination
Background
AI-powered search tools can generate plausible-sounding but fabricated claims, fake citations, or misattributed findings, a phenomenon known as hallucination. In academic research, this is particularly dangerous because users may incorporate inaccurate information into their work. This question assesses how frequently the tool produces verifiable errors across multiple test queries. To score this consistently, run a minimum of 5 distinct test queries: (1) a broad conceptual query in your discipline, (2) a highly specific niche query where you know the literature, (3) a query seeking recent research published post-2024, (4) a negative query designed to test boundaries (e.g., asking about a fabricated concept to see if the tool flags it as non-existent), and (5) a synthesis query asking the tool to summarize three known papers. Test with domain-specific queries where you can verify claims against known literature. This question only applies to AI-powered tools. Note: this question focuses on factual correctness of outputs. It is distinct from US2 (Critical thinking prompts), which evaluates whether the tool actively supports user verification and critical evaluation, regardless of whether the underlying claims are correct. A tool can score poorly on RE1 (factual errors) but well on US2 (good verification prompts), or vice versa. This question should be marked N/A if the tool functions purely as a traditional, non-generative search engine that only retrieves and links to existing publications without synthesizing summaries or generating text responses.
Examples
- 0
- When asked about the effectiveness of a specific drug treatment, the tool cites a 2023 Nature paper by 'Johnson et al.' that does not exist. Cross-referencing three other claims reveals that two of the five cited papers are fabricated, and one real paper is cited with an incorrect conclusion.
- 1
- The tool correctly summarizes a body of literature on climate feedback loops but attributes a key finding to the wrong author (citing "Martinez 2023" instead of "Martinez 2022") and states the effect size is "3.2°C" when the paper reports "2.1°C." Most citations resolve to real papers, but cross-referencing reveals recurring factual errors in names, years, and statistics.
- 2
- In a response about dietary fats, the tool generates a "References" list containing 4 real, verifiable papers with correct DOI links and accurate author names. However, the output presents the claim "Saturated fat has no effect on heart disease" as settled medical consensus, omitting that this finding comes from a single meta-analysis whose conclusions remain actively contested in the literature.
- 3
- When prompted about CRISPR efficacy, the tool outputs verifiable claims with accurate citations, such as correctly quoting the 2021 Doudna paper. The UI's inline citation popover resolves to real papers, faithfully representing findings and clearly noting ongoing debates regarding off-target effects.
Evidence (1)
Capture2026-06-19 12:37 - RE22
Across repeated identical queries, the tool cites the same top sources and reaches matching conclusions, while secondary citations and specific sentence phrasing shift between individual generated runs.
see two evidence captures for a comparison between two runs with highly similar imput queries. The match was pretty good, but there are some nuanced differences. The differences in the results can directly be explained by the different interpretation of the user's query.
RE2DetailsOutput consistency
Background
A reliable research tool should produce substantively equivalent answers when given the same query multiple times. High variance (where the same question yields different conclusions, sources, or recommendations on successive runs) undermines trust and makes the tool unsuitable for systematic work. This question evaluates reproducibility by asking the reviewer to submit identical queries and compare the substance of the responses. Small surface-level differences in phrasing are acceptable; substantive differences in claims or conclusions are not. For deterministic, index-based databases, identical queries should yield identical results. Score 3 if result lists are perfectly stable. Score lower if results fluctuate between repeated runs without a documented update to the underlying index database, which may indicate unstable scraping, caching issues, or unpredictable API routing. Mark this question as N/A if the tool does not accept user queries, requires entirely distinct inputs for each session, or is explicitly designed as a single-use, non-generative utility where repeated iterations are functionally impossible.
Examples
- 0
- When the user clicks the 'Generate' button three times for the same query, the output box displays contradictory conclusions: "strong evidence," "mixed evidence," and "limited evidence." Each run cites entirely different papers, such as "Smith 2023" in run one versus "Jones 2024" in run two.
- 1
- The same query run five times mostly agrees on the conclusion ('moderate evidence') but in two runs, the tool recommends a specific meta-analysis that does not appear in the other three runs. The user would need to re-run the query to feel confident about the result.
- 2
- Across three runs, the tool consistently identifies the same three key studies and reaches the same conclusion. The phrasing varies slightly ('strong evidence' vs. 'compelling evidence') and one run includes an additional secondary source, but the core claims are stable.
- 3
- Running the query "Does metformin reduce mortality?" five times yields identical substantive outputs. All five runs cite the exact same five papers (e.g., the 2021 Cochrane Review), report "a 22% relative risk reduction," and display the same conclusion in the summary panel.
Reviewer note
see two evidence captures for a comparison between two runs with highly similar imput queries. The match was pretty good, but there are some nuanced differences. The differences in the results can directly be explained by the different interpretation of the user's query.
Evidence (2)
Capture2026-06-19 12:42 
Capture2026-06-19 12:42
Usability●●○2.5/3
- US12
Supports structured export in at least one standard format (BibTeX, RIS, or CSV) that imports cleanly into reference managers, but lacks direct integrations, API access, or browser extension support.
No reviewer note
US1DetailsWorkflow integration
Background
Research is a multi-tool workflow: search results need to flow into reference managers, citation tools, and writing environments. A tool that traps results in a silo (with no export or integration options) creates friction and discourages systematic use. This question evaluates whether and how well the tool connects to established academic workflows. Look for export formats (BibTeX, RIS, CSV), direct integrations (Zotero, EndNote, Mendeley), API access, or browser extension support. This question is N/A if the tool does not generate or manage individual search results or citations (e.g., a writing environment, flashcard app, or grammar checker) that would need to be exported to a reference manager.
Examples
- 0
- There is no export button, no download option, and no API documentation. The only way to save results is to manually select and copy text from the browser, then paste it into another application.
- 1
- Using the browser's standard text selection tool, users can copy a displayed article title like "AI in 2024". However, without a dedicated structured export button, the user must manually re-enter this text and author details into their Zotero reference manager.
- 2
- The tool has an 'Export' button that downloads selected references as a BibTeX or RIS file, which can be imported into Zotero or EndNote. No direct integration or browser extension is available.
- 3
- Clicking the Zotero connector icon in the browser toolbar automatically captures the displayed references with full metadata. The tool also offers a REST API with documentation for programmatic access, and supports direct export to EndNote, Mendeley, and BibTeX.
Evidence (1)
Capture2026-06-19 12:36 - US23
The tool actively prompts verification: confidence scores per claim, flags for contested findings, side-by-side source comparison views, or built-in fact-checking aids that guide the user to evaluate rather than passively accept.
The tool clearly shows the thinking and collection process, and also indicates confidence of a match per-item in a clear manner, including additional evidence from citing papers.
US2DetailsCritical thinking prompts
Background
AI search tools that present outputs as authoritative facts (without caveats, source visibility, or verification prompts) risk encouraging passive acceptance. This is particularly concerning in academic settings where critical evaluation of sources is a core skill. The best tools actively support the user's judgment by surfacing original source material, flagging uncertainty, providing confidence indicators, or offering built-in comparison tools. This question only applies to AI-powered tools. Note: this question complements RE1 (Accuracy and hallucination). While RE1 evaluates whether the tool's factual claims are correct, this question evaluates whether the tool actively supports the user in verifying and critically evaluating those claims, regardless of accuracy. A tool with perfect factual accuracy still scores low here if it presents claims without any verification aids. Select N/A if the tool operates as a traditional, non-generative search engine that merely directs users to external webpages without generating synthesized summaries.
Examples
- 0
- The tool generates a paragraph summarizing research findings with no source links, no disclaimer, and no indication that the text is AI-generated. It reads like an authoritative encyclopedia entry.
- 1
- A gray disclaimer banner below the prompt box displays the text: "AI outputs may be inaccurate." Although the tool generates a 300-word summary of the "2023 Market Report," no in-text citations or hyperlinks are provided, leaving the user unable to verify the specific statistics.
- 2
- When the AI states "sleep improves memory," inline citation [3] expands into a side-by-side panel. The panel displays the PDF of the 2022 Walker study and highlights the exact sentence, "consolidation of declarative memory," showing the specific extracted text alongside the generated synthesis.
- 3
- In addition to showing original source excerpts, the tool displays a confidence score for each claim (e.g., 'High confidence — supported by 3 independent studies'), flags claims where sources disagree ('This finding is contested — see alternative view'), and includes a 'Compare sources' button that opens a side-by-side view of all referenced papers.
Reviewer note
The tool clearly shows the thinking and collection process, and also indicates confidence of a match per-item in a clear manner, including additional evidence from citing papers.
Evidence (1)
Capture2026-06-19 12:38
Soundness●●○2/3
- SE12
Test queries surface at least some non-English or non-Western sources where relevant to the search topic, OR documentation specifies concrete methods used to reduce demographic or geographic bias in retrieval.
No reviewer note
SE1DetailsBibliographic equity & diversity
Background
Search tools can systematically favor certain perspectives, languages, regions, or publication venues, often reflecting biases in their training data, indexing sources, or ranking algorithms. In academic search, this can manifest as over-representation of English-language, Western-published, or high-impact-journal sources at the expense of research from the Global South, non-English publications, or smaller venues. This question evaluates whether the tool's index and ranking produce equitable, geographically diverse results. Test with queries that should return diverse geographic and linguistic results. This question is not applicable if the tool is a highly specialized, closed-domain database containing only a single known source or geographic focus (such as an internal institutional repository or a narrow legal database) where bibliographic diversity is structurally impossible.
Examples
- 0
- Searching for "agricultural adaptation strategies" returns exactly 50 results exclusively from the US and EU. The 20-page "Technical FAQ.pdf" omits the phrase "source diversity," and the UI's uneditable "Top Results" panel exhibits zero sources from Sub-Saharan Africa or South Asia.
- 1
- The tool's 'About' page states 'We strive for diverse and representative search results' but provides no data on source distribution, no documentation of mitigation techniques, and test queries still return predominantly English-language Western sources.
- 2
- A test search for "traditional medicine" surfaces the "African Journal of Traditional Medicine" alongside Western journals. The technical docs state: "a re-ranking step promotes geographic diversity." However, this page lacks quantitative metrics showing the actual geographic distribution of indexed sources.
- 3
- The tool's "Global Voices" filter lets users limit results to the Global South. Testing the query "history of mathematics" shifts results from 90% US/UK sources to include Iran and India. The published 'Algorithmic Debiasing' documentation details the counterfactual data balancing used in the ranking algorithm.
Evidence (1)
Capture2026-06-19 12:33 - SE22
Clearly states concrete retention periods and encryption practices (e.g., 'queries retained for 30 days', 'AES-256 at rest, TLS 1.3 in transit') so reviewers can verify compliance with institutional data governance policies.
No reviewer note
SE2DetailsData handling practices
Background
Beyond the basic privacy policy (covered by the quality gate), this question evaluates the depth and specificity of the tool's data handling practices. Researchers need to know exactly how long their data is retained, whether it is encrypted, what happens in case of a breach, and whether they can export or delete their data. Strong data handling practices are especially important for tools used in institutional settings where data governance policies may apply. Look for specifics, not just 'we protect your data' but actual retention periods, encryption standards, and breach notification timelines. Note: this question extends the data privacy quality gate (QG1), while QG1 is a pass/fail check on whether the vendor commits to not selling or sharing user data, this scoring question evaluates the depth and specificity of those commitments (retention periods, encryption standards, breach protocols). This question is not applicable if the tool operates entirely offline on local infrastructure and is designed so that no user data is ever collected, transmitted, or stored by the provider.
Examples
- 0
- The 'Privacy Policy' webpage features a single section titled 'Our Commitment', quoting verbatim: 'We take data protection seriously.' It completely lacks retention periods, encryption methods, breach protocols, or any mechanism for users to access or delete their data.
- 1
- The privacy policy includes a section on data protection that says 'Your data is stored securely and protected with encryption' but does not specify the encryption standard, how long data is kept, or what happens if a breach occurs.
- 2
- The tool's documentation states: 'Search queries are retained for 30 days and then permanently deleted. All data is encrypted at rest using AES-256 and in transit using TLS 1.3. Users can request data deletion via their account settings.'
- 3
- A comprehensive 'Data Handling' page covers: retention periods per data type (queries: 30 days, account data: until deletion request), encryption standards (AES-256 at rest, TLS 1.3 in transit), breach notification within 72 hours per GDPR, data residency (EU servers for EU users), and a one-click 'Export my data' and 'Delete my data' feature in account settings.
Evidence (1)
Capture2026-06-19 12:22
Traceability●●●3/3
- TC13
Citations link to the exact passage, page, or highlighted paragraph in the source document that supports the claim; the user can verify the claim in one click without searching within the paper.
No reviewer note
TC1DetailsSource attribution depth
Background
The depth of source linking determines how efficiently a researcher can verify claims and locate original material. A tool that links to a journal homepage requires the user to manually search for the specific article; a tool that links directly to the relevant passage saves significant time and enables precise verification. This question evaluates the granularity of citation links, from broken links (worst) to paragraph-level deep links (best). Test by clicking citations in the tool's output and noting where they lead. This question is N/A if the tool's output does not include any citations or source references to evaluate.
Examples
- 0
- The tool shows reference numbers in its output, but clicking them does nothing; there are no hyperlinks. Alternatively, the links exist but resolve to 404 error pages or redirect to the tool's own homepage.
- 1
- Clicking the superscript "[1]" citation link redirects to nature.com instead of the specific paper. To find the text "quantum entanglement," the user must independently search the journal's homepage. The system fails to resolve the DOI for the source document.
- 2
- Each citation links directly to the article's abstract page on the publisher's site (e.g., doi.org/10.1038/s41586-023-12345). The user can immediately access the paper's metadata and abstract.
- 3
- Citations link to the exact passage within the source document that supports the claim, for example, a specific page in a PDF or a highlighted paragraph in an open-access article. The tool uses RAG (Retrieval-Augmented Generation) with passage-level attribution.
Evidence (2)
Capture2026-06-19 12:42 Capture2026-06-19 12:38 - TC2N/A
Not applicable
The source database takes care of this: filtering out specific item types and handling retracted data. It is not included in this tool.
TC2DetailsSource quality indicators
Background
Not all academic sources are equal: papers may be retracted, published in predatory journals, or exist only as non-peer-reviewed preprints. A tool that treats all sources identically (without any quality indicators) makes it harder for researchers to assess the reliability of the evidence behind a claim. This question evaluates whether the tool provides contextual quality metadata such as retraction status, peer-review labels, citation counts, and publication type. Look for quality badges, labels, or metadata near each cited source. Note: this question evaluates item-level metadata enrichment, whether the tool warns the user about the status of individual papers during search (e.g., retraction notices, publication type labels). It is distinct from TR1 (Data source clarity), which evaluates macro-level scope transparency about what the tool's index contains. Mark this question as N/A if the tool does not retrieve or cite individual academic sources (such as papers or preprints), or if it functions exclusively as a generalized search engine without the capacity to extract and display specific publication metadata.
Examples
- 0
- The tool cites a paper that was retracted in 2022 due to data fabrication, with no indication of the retraction. It also includes sources from a known predatory publisher on Beall's list, presented identically to legitimate journal articles.
- 1
- All cited sources appear with the same formatting and no quality labels. There is no distinction between a Nature peer-reviewed article, a preprint on arXiv, and a blog post; all look equally authoritative in the tool's output.
- 2
- Each source is labeled as either 'Peer-reviewed article', 'Preprint', 'Conference paper', or 'Thesis'. The tool filters out known predatory journals but does not show citation counts or retraction status.
- 3
- Each cited source displays: a 'Peer-reviewed' or 'Preprint' badge, the citation count from Semantic Scholar, a retraction warning flag (e.g., 'This paper was retracted on 2024-01-15'), an 'Open Access' icon when the full text is freely available, and the journal's impact factor quartile.
Reviewer note
The source database takes care of this: filtering out specific item types and handling retracted data. It is not included in this tool.
Additional Evidence
